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ABSTRACT 

Two papers are presented from a panel discussion 
moderated by Naomi Zigmond, who introduces the papers with a note on 
definitions and measures of intervention effectiveness. "Some 
Thoughts on Effective Intervention for Handicapped Preschoolers," by 
Phillip strain, notes that special education researchers attempt to 
be efficient, economical, politically compatible, and consistent with 
values, and later consider effectiveness only in a belated fashion. 
Researchers should consider how consumers evaluate effectiveness, 
because interventions will not be used if people cannot see a 
difference in child outcomes attributable to these interventions. 
Researchers should also recognize the importance of longitudinal 
studies. The second paper, "Markers of Effectiveness at the Secondary 
Level in Special Education" by Eugene Edgar, contends that the 
important aspect of secondary special education is to prepare 
students with skills and attitudes that will enable them to 
experience some quality of life, and to offer them the opportunity to 
practice their skills in the natural environment. Measurement of 
student outcomes is very difficult and involves issues of how to 
obtain data, when to obtain data, and from whom. The paper examines 
follow-up studies of special education school leavers and offers 
guidelines for conducting future effectiveness research. (JDD) 
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Panel Discussion: 



What is Effectiveness? 



Naomi Ziginond^ Moderator: One of the most critical questions facing 
practitioners and researchers in special education concerns the definition 
of effectiveness. What is special education, for? What are indicators that 
special education services have been effective? Should we^ as a field, be 
satisfied with very narrow definitions of effectiveness, i.e., positive 
changes in a single behavior which has been the target of instruction? Or, 
do we expect special education services, especially those designed for 
mildly handicapped student to remedy the students* problems, make the 
students all better? V7e have all faced this issue, as practitioners and as 
researchers: for example, a master's student of mine, after some years out 
in the field, came back and told me about her experience with learning 
disabled students. She had worked relentlessly to have these students 
achieve some success in academic work, and at the end of a couple of years, 
most of her students had done remarkably well. When she proudly displayed 
the data to her supervisor, the supervisor said, "Well, they probably 
weren't learning disabled to begin with!" Clearly, the supervisor's view of 
effectiveness was not that the students would be "cured." If a teacher 
could accomplish that, the students must have been misdiagnosed! 

The issue came up again in some of the work that Helen Thornton and I 
have been doing on dropouts. We locate high school earning dr.sabled 
graduates and dropouts a year or so after what should have been 12th grade. 
We ask these young people to take a basic skills test and we inquire about 
their employment status and employment history. Vlhen we look at the basic 
skills levels of learning disabled students who have gradurted from high 
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school, we find that they are still very far inferior to the levels of 
control peers. Does that mean that the special education program given to 
these LD students had not been effective? Should we have expected special 
education services to narrow the achievement gap? On the other hand, the LD 
graduates were employed at rates that were equal to,, and at pay rates that 
were equal to, non-disabled peers who had also graduated from high school. 
Did that mean that their special education program had actually been a 
success? Should we expect that there would be some life-long penalty for 
being learning disabled? After all, LD was a condition these young people 
carried with them even into the employment market. Or are effective special 
education services ones that produce no long-term penalty? 

Well,, I brought these sort of ill-formed questions about how to measure 
the effectiveness of special education to the Planning Committee last 
December, and the outcome was this panel this morning: a - discussion of 
"effectiveness" in special education^ and more specifically, since many of 
us are involved in intervention research, a discussion of appropriate 
measures of successful interventions, i.e. appropriate measures of 
effectiveness? We have with us Phil Strain from the University of 
Pittsburgh, Joe Jenkins from the University of Washington, and Eugene Edgar 
from the University of Washington. Each one, in turn, will give us their 
views on defining the outcomes of special education, one at pre-school, 
elementary, and secondary and post-secondary levels. They will address what 
special education trying to accomplish, and how do we measure that? We're 
going to go in my favorite order, reverse alphabetical, which means we start 
with Strain, and then Jenkins, and then Edgar. We'll hold questions until 
the end when I hope we will have time for an interactive discussion. 
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I want to begin by pointing out that being effective, as in 
helping children to learn important things about themselves and 
the world around them, is a small piece of what it takes to move 
empirical findings from the literary museums we call journals to 
everyday practice. We in the research community have a much more 
broad, difficult, and occasionally improbable agenda* As 
indicated in Figure 1, we must also be efficient, economical. 



Insert Figure 1 Here 



politically compatible, consistent with values and, of course, we 
have to look to the outside world. 

I have also tried to suggest in this Figure that the 
adoption of educational practices often " follows a path that 
belatedly considers effectiveness. And, I have contrasted this 
path with a more typical one for medical procedures, where 
effectiveness is always the initial concern. Finally, let me 
suggest that many educational researchers see the world quite 
differently from those who will or will not adopt their effective 
procedures. We have, I think, made some honest attempts in 
recent years to be efficient, economical, compatible, and all the 
rest. I think we should continue to pursue that course, with 
this caveat: That we nail the question of effectiveness prior to 
working on other standards for adoption. I also think we should 
go about studying how to make effectiveness a more valued 
dimension in the eyes of classroom teachers, school 
administrators, and tax payers. 
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The second point I would like to raise is very much related 
to the first. The point has to do with using empirical methods 
to understand how our various consumers evaluate effectiveness. 
Figure 2 depicts some of the relevant concerns around consumer 



Insert Figure 2 Here 
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issues. At the most basic leveT"we need to know if the opinions 
of consumers are positive, neutral, or negative in regard to 
effective intervention practices. We know enough at this point 
to say that effective interventions will not be used if people do 
not like them, or, if they cannot see a difference in child 
outcomes attributable to these interventions. It is interesting 
to me that this phenomenon is so widespread. It applies to the 
use of aversive procedures to treat self-injury, the full litany 
of behavior management procedures applied to behavior problem 
children, self -monitoring with adolescent offenders, and 
integrated service delivery for preschoolers. 

At a slightly more complex level we need to determine the 
relationship between opinions and specific outcomes. For 
example, does a unit of child behavior change yield a unit of 
parent or teacher satisfaction? Do we have to produce the 
behavior change for a long time for people to see it? What are 
people really happy about? Is it child behavior change or the 
intervention experience itself? Of course, to answer any of 
these questions we have to decide that what people say about our 
interventions is important and valid. If we launch such studies 
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to answer these questions on a wide-scale basis then we might 
eventually reach the final goal of identifying levels of child 
behavior change that predict good consumer opinions, and 
therefore predict a good likelihood of continued intervention 
use • 

My final point regarding effectiveness reflects a strange 
form of amnesia that overcomes researchers of every theoretical 
persuasion who are fortunate enough to have any positive, long- 
term follow-up data. We forget about the contemporary 
environment while giving superordinate credit to historical 
events. I think this f orgetf ullness is a problem because it 
obscures an analysis of why we continue to be effective, or not. 
In early childhood work at least, what happened as much as 20 
years ago^is somehow directly causative of the most complex, 
interactive, and elusive of human behavior. As a mild aversive, 
let me point out that the only theory which is consistent with 
such a historical attribution is psychoanalytic, pre Anna Freud. 
While the interactionists may argue that they have derived a 
conceptually believable scheme to handle two decades of person- 
environment transitions, there are too many unknown steps in the 
sequence to make me very comfortable with such an approach. 

As an alternative to looking solely to ^^,he past, or to 
intervening and as yet non-specified events, let me offer a 
simple minded example of how follow-up data can be considered as 
influenced by the contemporaneous environment; influenced by, 
therefore controllable by, and predictable. 



ERIC 



9^. 91 



Consider the illustrative data in Figure 3. Each panel 
represents a different grade level (K-2) and each data point 



Insert Figure 3 Here 



within panels represents one days' data collection. The 
relationship between on- task behavior on the part of the former 
early intervention recipient and class structure is suggestive, 
and these correlational findings set the stage for the subsequent 
f\inctional analysis of on-task behavior as a follow-up measure. 

I am not suggesting in this Figure that all follow-up 
measures are so fluid. I am suggesting, however, that one need 
not rely solely on history to explain variance in follow-up data. 
More importantly, I am suggesting the possibility that "being 
effective" over the long-haul may well imply longitudinal 
intervention. Trying to define effectiveness is a bit like 
trying to throw a fastball soaked in 40-weight oil. The harder 
you try, the less likely it seems that success will come your 
way. At least part of the fastball and definitional problems 
come, I think, from persuing a well-travelled and fruitless road 
one too many times. We need a rather radical set of 
alternatives. Try as we might, we cannot throw our fastball 
soaked in oil and we can not grasp effectiveness with a singular 
focus on immediate child outcomes. Our focus also needs to 
include longitudinal outcomes and a careful analysis of 
contemporaneous environments. Our focus also needs to include 
the opinions of consumers, sampled rigorously and analyzed 

ERIC 1 2 92 



accordingly. Finally, our focus needs to include the full-range 
of dimensions that influence school practices and child outcomes* 
Efficient, economical, and politically compatible are a few of 
the other adjectives that must apply to our intervention 
procedures. 
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There is a point on the Willamette River near Eugene, 
Oregon, where a classical Oregonianism is to be found: 
"Warning, dangerous rapids ahead, only expert kayakers or fools 
should proceed*" Attempting to discuss the effectiveness of 
secondary and postsecondary programs in special education is no 
easy matter* Even developing a framework by which to 
conceptualize an evaluation schema is fraught with philosophical 
rapids and methodological whirlpools. To venture past this 
initial point of conjecture requires expertise that exceeds my 
current white water rating. However, never being one who lacks 
fool hardiness, I will slide on by this starting point. 

Following Naomi's lead," I have organized my thoughts around 
her four basic questions: 

1) What is special education trying to accomplish? 

2) How can these outcomes be measured? 

3) What are the indications that special education 
programs have been effective? 

4) From a research perspective, what are the "rules of 
evidence" to support a contention that special 
education is effective? 

What is Special Education Trying to Accomplish? 

This is obviously the most crucial point of the entire 
exercise. What is the purpose of special education, or indeed 
of public education in general? The easy answer for us today is 
to accept the "OSERS Bridge" model and declare that special 




education is "an outcome oriented process encompassing a broad * 
array of services and experiences that lead to employment" 
(Will, 1984, p. 1). I personally reject that premise as being 
far too simplistic. 

Chester Finn (current Assistant Secretary for Research and 
Improvement in the Department of Education) views the outcomes 
of schools in terms of preparing students (all students, he 
argues) with skills for 1) the social system in which we live, 
2) personally fulfilling lives, and 3) the next phase of their 
lives, be it employment or higher education (Finn, 1986). 

A third view is that of Wehlage (1983), who has developed 
proposed outcomes based on adolescent developmental theory. He 
advocates teaching coping skills (self-management, conflict ' 
resolution, and problen solving) as well as specific skills. 

Thus, we desire our students to be job-ready for employment, 
knowledgable of our political system, self-assured as to "who 
they are," competent in reading, socially adept, emotionally 
intact... truly ready to be productive contributors to our 
society. 

These definitions of desired outcomes ring more true to me 
than simply advocating for employment. But this stretch of 
white water is lethal. Philosophically, we (in the United 
States) view the public education system (K-12, or 0-21 for us 
in special education) as the "great entitlement," the process by 
which each of us is given the opportunity to partake in our way 
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of life. Education is the religion of democracy (Boorstin, 
1974), The converse is that, other than the public school 
system, our society provides scant assistance to its citizens. 
Public schools are "it" as far as opportunities go for receiving 
services from our government (entitlements, assistance, help, 
compassion). For many of our citizens, and especially those 
from the underclass, public schools are their only chance to 
acquire a reasonable shot at having a minimally acceptable 
quality of life. Given this enormous importance of public 
education, the debate on desired outcomes for special education 
students at the secondary level must not stop with easy 
solutions such as "employment" or esoteric cliches such as 
"productive contributors to our society." 

I contend that the important aspect of secondary special 
education is to prepare our students with skills and attitudes 
that will enable them to experience some quality of life (have 
choices, experience joy, interact with friends, feel productive, 
care for their personal needs, compete in our society). I also 
believe the outcome of schooling should include opportunities to 
practice these skills in the natural environment. Thus, 
experience is also an outcome. I also believe that the 
education system must assume the role of identifying those 
students who will require ongoing support and assist those 
individuals in locating needed services. Finally, I believe 
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educational professionals have the ethical duty to inform their 

fellow citizens of the human service needs of their students. 

Now, I am the first to acknowledge that my list is no better 

(nor worse, I will add) than those proposed earlier. This is a 

thorny issue (how many metaphors ^an I use?). My final 

recommendation is that this debate become a valued part of our 

"scientific endeavors." I know, and agree to a large extent, 

that we need to base our discussions on data, that our journal 

articles need to be data-based, empirical inquires into the 

nature of nature herself. Yet, are we asking the right question? 

"It is axiomatic in science that progress hinges on asking 
the right question. Surprisingly, once the right question 
is asked the answer seems almost to tumble forth. That is a 
retrospective view; in prospect, it takes genuine (and 
mysterious) insight to see correctly into the brambles 
created by previous ill-chosen verbalizations" (Hardin, 
1978, p. 29). 

We, as a subgroup of our profession, should advocate for 
open debate on "the purpose of special education," 
How Can These Outcomes Be Measured ? 

How does one measure quality of life? Or the presence of 
skills and attitudes necessary to freely partake of "our way of 
life" without resorting to concrete, objective facts such as 
salary level and place of residence, or subjective self-report 
statements ("I'm satisfied with my life")? These are difficult 
questions we must all confront and debate. 

Conducting follow-up studies of special education school 
leavers poses many problems. Even after the outcomes have been 
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defined, a major issue remains: who provides the 
information— former students, parents, a third party? For some 
questions, the answer is clear — i.e.. How satisfied are you 
with your life? For others, however, there is no clear 
method— -i.e.. What is your child's current employment status? 
Sometimes the parents will not know the employment status of 
their adult child, sometimes the former student will not tell 
the truth, and locating knowledgeable third-party informants is 
often very difficult. In other instances, the former student is 
not able to communicate. Selecting informants is clearly a 
difficult task. 

Another issue to consider is how to obtain the data. The 
most cost-effective procedure is the telephone interview. This 
practice may not be as efficient as in-person interviews wnere 
cost, however, is very high. Written questionnaires seem least 
desirable and often result in a biased sample as well as a low 
return rate. 

When to probe is also an issue. ' Most studies report data 
sometime during the first year after leaving high school. 
However, we need more data about the lives of former students 
during the years after exit. Just how long to track these 
students is not clear. There is some evidence that all American 
youth "flounder" for the first several years after high school 
(Hamilton, 1986). If this is true we need to follow our 
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graduates for 3-4 years after graduation mainly to determine 
their eventual status in life. 

Finally, there is the issue of quality. Regardless of how 
many minutes I spend thinking about how to measure quality, I 
always return to ethnographic procedures. The work of Andrea 
Zetlin and Kike Murtaugh (1987) provides examples of the type of 
data that can be obtained using these procedures. Problems with 
small Ns and reliability of measures notwithstanding, 
ethnographic methods yield information on quality issues (i.e., 
type of friendships, extent- or opportunities) that can be 
obtained in no other way. 

What Are The Indications that Special Education Programs Have 
Been Effective ? 

There have been numerous follow-up studies of special 
education school leavers (graduates, age-outs, and dropouts) 
conducted in the 1980s. These studies have used parents or 
students as informants, have sampled students at 1-5 years from 
the point of leaving school, and have focused almost entirely on 
job status, postsecondary education, and living situation. 
There is a fairly consistent pattern of results. 

Overall, approximately 60% of the special education 
graduates are employed (Mithaug, Horiuchi, & Fanning, 1985). 
For students with more severe disabilities, such as moderate 
retardation, the employment rate is lower (41%) (Wehman, Kregel, 
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& Seyfaith, 1985), while close to 70% of the LD students are 
employed (Zigmond & Thornton, 1985). 

The data vary considerably in regard to postsecondary 
education* Hearing and visually impaired students have a high 
rate of attendance (60%) as compared to severely behv^vior 
disabled students (23%) (Edgar & Levine, 1987). The real test 
of postsecondary attendance, of course, is graduation rates, for 
which there are few data. 

Most special education graduates live with their parents or 
relatives 2-3 years after leaving high school. Of course, this 
is also true of nondisabled youth. 

In addition, about 25% of all special education school 
leavers tend to earn more than the minimum wage and few receive 
any type of benefits (i.e., health insurance) (Hasazi, Johnson, 
Gordon & Hull, 1988). The data on nonhandicapped students 
appear to be very similar as to wages and benefits. Youth in 
America, in general, live in poverty, and have very poor health 
care. 

There are examples of special education graduates doing well 
after high school, but most of these students are receiving seme 
type of ongoing support services (Wehman, Hill, Goodall, 
Cleveland, Brooke, & Pentecost, 1982). 

A large percentage of mildly handicapped students never 
graduate from high school - they drop out. Data on dropouts 
from special education are difficult to determine but there is 
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con;:iderable evidence that many mildly handicanped students do 
not complete high school programs. Zigmond & Thornton (1985) 
report a dropout rate of 50% for LD students, while Hasazi, 
Gordon, & Hull (1985) report a rate of 35% for all special 
education school leavers. 

A final point ^s that of comparison to some norm or 
standard. Even after collecting all these data, to what 
standard do we compare our results? Certainly using a 
nonhandicapped cohort provides some comparison. But what about 
the iniquities that many of our nonhandicapped youth 
experience? If we can report that youth who pass through the 
special education system are no worse off than youth who have 
not been in special education, can we be satisfied? 1 think 
not. We must develop some concrete standard by which to measure 
our outcomes. 

Overall, I believe the data do not support the notion that 
"things are going well for special education graduates." 
Rules of Evidence 

I propose the following notions as guidelines for conducting 
future effectiveness research. Four points seem to be 
relevant: 1) operationalizing outcomes; 2) analyzing data by 
subgroups within the total special education population; 3) 
measuring the process variables (the independent measure); 4) 
attention to the size of Ns and national as well as local 
representation. 
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Outcome measures . Obviously, I question many of our current 
outcome measures. However, at a minimum, we should consider the 
following: skill levels (e.g., achievement test scores); success 
in postsecondary training (e.g., graduation from college or 
vocational school); job acquisition (salary levi benefits, 
promotions); friendships; reported satisfaction with life; and 
absence of negatives (e.g., legal problems, 
institutionalization, being unengaged). Our current data base 
consists almost exclusively of information on jobs, salary 
levels, and postsecondary education. We must expand these 
measures to include the more qualitative aspects of life. 

Subgroup analysis . The special education population is 
incredibly heterogeneous. I strongly recommend careful analysis 
in terms of the following subgroups: 

1) Type and severity of disability ^ Even with the 

controversy concerning inappropriate labeling, we cannot 
squash together various types of disabilities. We must 
give careful attention to levels of severity, especially 
in regard to mental retardation. As obnoxious as IQ 
scores may be, indiscriminate groups of "MR" are even 
more annoying. Outcome studies need to identify the 
populations as finitely as possible and report data in 
terms of subgroups. Data can always be aggregated; 
seldom can they be divided. 
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2) Gender . There are considerable data to support the view 
that gender influences outcomes. Within each disability 
category, researchers should consider gender as an 
important factor. 

3) Ethnicity . Regardless of the reasons, ethnic minorities 
are -overrepresented in the special education 
population. We must analyze our results by ethnicif • so 
we do not miss possible trends. Have no doubts, if we 
do this, we are aj2 going to feel uncomfortable. 

4) Social class . There is reason to believe outcomes are 
directly affected by the level of the family social 
status of our students. Students from underclass 
families do less well than students from the middle 
class. Even though this marker is most difficult to 
obtain, we must biflin to analyze our results by social 
class. I fear that social status accounts for the major 
amount of all variance in our results. 

Measuring the process variables . Simply collecting data on 
the postschool status of special education students is not- 
sufficient. We must begin to obtain follow-along data on the 
types of programs students receive while in school and correlate 
the programs to outcomes. The inpact of integrated versus 
segregated programs on student outcomes is one type of issue to 
address. In addition, we need tc follow students who have 
received systematic vocational training as compared to those who 
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receive only academic training. The issue here is that of 
programs which offer sequential vocational experience, not 
simply attendance in a "world of work course." Data on 
attendance and skill acquisition while in the secondary program 
are needed. The only way to do this is to implement systematic 
data collection procedures while students are still in high 
school, and follow these students throughout their high school 
careers and into the adult world for several years. This would 
truly be a follow-along study. 

Population considerations .' As researchers conduct their 
studies, they must take care to collect data using large samples 
that represent various geographical regions. Urb^n, suburban, 
and rural areas need to be sampled as well as students from 
industrial and agricultural communities. 

Many current studies have trouble locating significant 
numbers of special education school leavers (usually about 60% 
are contacted). The most mobi-le students tend to be missed. My 
guess is that the students who are difficult to locate probably 
are experiencing less success than those students who are 
discovered with the exception of the Iowa study (Sitlington, 
1987) the "hit rate" of most studies is suspect. 
Summary 

Ed(»cation is the major entitlement for all citizens of the 
United States. Our educational system provides the process by 
which our youth enter the adult world, prepared to be happy, 
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productive citizens. Many students served in special education 
enter the adult world with a minimal likelihood of achieving a 
successful life. These students come to us not of their own 
free will but rather by default, when other options fail them. 
We must carefully celebrate our successes, and freely admit our 
shortcomirias. This problem demands bold, persistent 
experimentation. It is common sense to take a method and try 
it; if it fails, admit it frankly and try another (I paraphrase 
a quote by Franklin D. Roosevelt on announcing the New Deal, 
which is cited in Boorstin, 1987, p. 84). To do otherwise is to 
let down tiiose who place their confidence in us. We cannot be 
frightened off by the rapids in our path. 
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